-
Notifications
You must be signed in to change notification settings - Fork 73
ci: test gpu on self-hosted runners #108
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Conversation
I see that not only the job that I'm migrating to self-hosted runners, but also
Last month they were still passing on the default branch: https://github.com/filecoin-project/rust-filecoin-proofs-api/actions/runs/15213629917 @vmx Do you have an idea why that might be? cc @BigLep |
Things should work, but the current master patches the rust-fil-proofs crates to point to the master branch: rust-filecoin-proofs-api/Cargo.toml Lines 32 to 36 in b06f9fb
As there was a release of rust-fil-proofs, this repo should be updated to use the released version. So I suggest that the current maintainers update to those versions and the we'll see if things still fail. |
I added a release commit here - c5246a9 - and the workflow now passes. Now, the question is, what the release process is that we should follow? I see that the previous tags were created manually - https://github.com/filecoin-project/rust-filecoin-proofs-api/tags |
Thanks for the updates @galargh. @galargh : Would it maybe make sense to break this into two PRs (one for the release, and one for the CI adjustment)? @vmx : are there any steps we should follow for making releases (e.g., any |
The release commit in here had me confused, but I think we're just trying to do two separate things at once? I don't think I mind as long as it's not squash merged, but it would have been clearer for reviewing if they were separate PRs. My only question is about the nvidia drivers--are they already installed on the standard GitHub machines but not on the current AMI that we have access to? |
I would do the release separately. It's done similarly as for |
Yes, that was the plan all along. I just wanted to see whether the release fixes the build on self-hosted runners. Here's the separate PR that I just created: #109
The hosted runners do not have GPUs. On self-hosted ones, we do have dedicated GPUs - that's what we install the drivers for. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
This PR enables GPU testing on self-hosted runners by configuring the CI workflow to run on AWS g6e.2xlarge
instances with GPU support. The changes address the need to test GPU functionality that cannot be executed on standard GitHub-hosted runners.
- Migrates the test job from
ubuntu-24.04
to self-hosted runners with GPU capabilities - Adds CUDA driver installation for the self-hosted environment
- Updates package installation commands to use
apt-get
for better reliability
# https://www.nvidia.com/en-us/drivers/ | ||
- name: Install CUDA drivers | ||
run: | | ||
curl -L -o nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb https://us.download.nvidia.com/tesla/570.148.08/nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Downloading executable content over HTTP without integrity verification is a security risk. Consider adding SHA256 checksum verification after the curl command to ensure the downloaded file hasn't been tampered with.
curl -L -o nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb https://us.download.nvidia.com/tesla/570.148.08/nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb | |
curl -L -o nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb https://us.download.nvidia.com/tesla/570.148.08/nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb | |
# Verify SHA256 checksum | |
echo "b1e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2e2 nvidia-driver-local-repo-ubuntu2404-570.148.08_1.0-1_amd64.deb" > nvidia-driver.sha256 | |
sha256sum -c nvidia-driver.sha256 |
Copilot uses AI. Check for mistakes.
run: | | ||
sudo apt-get update | ||
sudo apt-get install --no-install-recommends --yes libhwloc-dev nvidia-cuda-toolkit ocl-icd-opencl-dev | ||
# TODO: Remove this and other rust installation directives from jobs running |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The TODO comment is incomplete and unclear. It should specify what should be removed and why, or what the complete sentence should be.
# TODO: Remove this and other rust installation directives from jobs running | |
# TODO: Once the AMI includes the Rust toolchain, remove this step and any other Rust installation directives from CI jobs to avoid redundant installations and speed up workflow execution. |
Copilot uses AI. Check for mistakes.
run: | | ||
sudo apt-get update | ||
sudo apt-get install --no-install-recommends --yes libhwloc-dev nvidia-cuda-toolkit ocl-icd-opencl-dev | ||
# TODO: Remove this and other rust installation directives from jobs running |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nitpick] Using a specific commit SHA for the action is good for security, but consider adding a comment explaining why this specific version is pinned, especially since the TODO above mentions removing rust installation directives.
# TODO: Remove this and other rust installation directives from jobs running | |
# TODO: Remove this and other rust installation directives from jobs running | |
# Pinned to a specific commit SHA for security and reproducibility. | |
# This version was chosen to ensure compatibility with the workflow; update only after verifying changes. |
Copilot uses AI. Check for mistakes.
- uses: dtolnay/rust-toolchain@21dc36fb71dd22e3317045c0c31a3f4249868b17 | ||
with: | ||
toolchain: 1.83 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this kind of sucks that we can't just use the rust-toolchain file for versioning, but note from https://github.com/dtolnay/rust-toolchain?tab=readme-ov-file#inputs about versioning:
Rustup toolchain specifier e.g.
stable
,nightly
,1.42.0
,nightly-2022-01-01
. Important: the default is to match the @Rev as described above. When passing an explicit toolchain as an input instead of @Rev, you'll want to use "dtolnay/rust-toolchain@master" as the revision of the action.
i.e. it wants you to use dtolnay/[email protected]
instead.
(I also notice other poeple are annoyed by this gap).
Related to filecoin-project/rust-fil-proofs#1775
Similar to filecoin-project/rust-fil-proofs#1785
This PR enables the job that requires running on a machine with a GPU. It will run on a
g6e.2xlarge
runner.